Eye gaze tracking method and apparatus and computer-readable recording medium

ABSTRACT

A method and an apparatus of tracking an eye gaze to determine where a gaze point of a user is located on a display unit of a device, based on a facial pose and a position of an iris center, are provided. The method includes detecting a facial feature in a captured initial image, three-dimensionally modeling the detected facial feature, tracking the three-dimensionally modeled facial feature in consecutively captured images, detecting an iris center in the consecutively captured images, acquiring an eye gaze vector based on the tracked three-dimensionally modeled facial feature and the detected iris center, and acquiring a gaze point on a display unit based on the eye gaze vector.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit under 35 U.S.C. §119(a) of a Koreanpatent application filed on Apr. 10, 2014 in the Korean IntellectualProperty Office and assigned Serial number No. 10-2014-0043205, theentire disclosure of which is hereby incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to eye gaze tracking methods andapparatuses and computer-readable recording media storing the eye gazetracking methods. More particularly, the present disclosure relates tomethods of determining where a gaze point of a user is located on adisplay unit of a device based on a facial pose and a position of aniris center.

BACKGROUND

An eye gaze tracking technique is a technique for tracking an objectseen by a user to detect a point to which a user's pupils are directed.Recently, the application of eye gaze tracking techniques has beenextended with the development of human interface techniques, as the eyegaze tracking technique is intuitive and convenient. basis

Recently, a video-based eye gaze tracking method based on imageprocessing has been widely used. The eye gaze tracking method uses amethod of acquiring an eye image through a camera and analyzing a gazedirection of an eyeball by image analysis. In this case, a scheme basedon a stereo a vision-based depth camera, an InfraRed (IR) camera, and anillumination device may be used for accurate eye gaze tracking.

However, the scheme based on an IR camera is not economical because itrequires additional expensive equipment in order to achieve satisfactoryquality.

Some companies (e.g., EyeTrackShop and GazeHawk) present an eye gazetracking method that does not use expensive equipment, but this eye gazetracking method has limitations on use and needs extensive calibration,and may not achieve sufficient accuracy.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present disclosure.

SUMMARY

Aspects of the present disclosure are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentdisclosure is to provide an apparatus and method for determining where agaze point of a user is located on a display unit of a device, based ona facial pose and a position of an iris center, and computer-readablerecording media storing the eye gaze tracking methods.

Hereinafter, basic aspects for a better understanding of the embodimentsof the present disclosure are provided in summary. This summary is notan extensive overview of all considered aspects and is not intended toidentify important or critical elements of all aspects and to accuratelydescribe the scope of some or all aspects. The purpose of this is tobriefly present some concepts of the aspects as the introduction of thedetailed description of the embodiments of the present disclosure.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the present disclosure, an eye gazetracking method is provided. The method includes detecting a facialfeature in a captured initial image, three-dimensionally modeling thedetected facial feature, tracking the three-dimensionally modeled facialfeature in consecutively captured images, detecting an iris center inthe consecutively captured images, acquiring an eye gaze vector based onthe tracked three-dimensionally modeled facial feature and the detectediris center, and acquiring a gaze point on a display unit based on theeye gaze vector.

In accordance with another aspect of the present disclosure, an eye gazetracking apparatus is provided. The apparatus includes a facial featuredetecting unit configured to detect a facial feature in a capturedinitial image, a facial feature tracking unit configured tothree-dimensionally model the detected facial feature and to track thethree-dimensionally modeled facial feature in consecutively capturedimages, an iris center detecting unit configured to detect an iriscenter in the consecutively captured images, and an eye gaze trackingunit configured to acquire an eye gaze vector based on the trackedthree-dimensionally modeled facial feature and the detected iris centerand to acquire a gaze point on a display unit based on the eye gazevector.

In accordance with another aspect of the present disclosure, acomputer-readable recording medium configured to store computer-readableinstructions that, when executed by a computer, perform theabove-described eye gaze tracking method.

The above aspects will be fully described later and will includefeatures stated in the claims. The following descriptions and theaccompanying drawings present aspects in more detail. However, theseaspects represent some of various methods in which the principle ofvarious aspects may be used, and the described aspects should beconstrued as including all of such aspects and equivalents thereof.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will become more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 illustrates an eye gaze tracking apparatus according to anembodiment of the present disclosure;

FIG. 2 is a flowchart of a method for acquiring a gaze point of a useron a display unit according to an embodiment of the present disclosure;

FIG. 3 is a flowchart of an eye gaze detecting method according to anembodiment of the present disclosure;

FIG. 4 is a continuation flowchart of FIG. 3, further illustrating theeye gaze detecting method according to an embodiment of the presentdisclosure;

FIG. 5 illustrates an example of acquiring a gaze point by an eye gazevector based on an eyeball center and an iris center according to anembodiment of the present disclosure;

FIG. 6 illustrates an example configuration of an eye gaze trackingapparatus according to an embodiment of the present disclosure;

FIG. 7 illustrates an example operation of the eye gaze trackingapparatus according to an embodiment of the present disclosure;

FIG. 8 illustrates an example in which a relative position between aneyeball center and a face does not change even when the face movesaccording to an embodiment of the present disclosure; and

FIG. 9 illustrates an example of an iris edge having a verticaldirection according to an embodiment of the present disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the present disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thepresent disclosure. In addition, descriptions of well-known functionsand constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of the presentdisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of the presentdisclosure is provided for illustration purpose only and not for thepurpose of limiting the present disclosure as defined by the appendedclaims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings, wherein like referencenumerals refer to like elements throughout. In this regard, the presentembodiments may have different forms and should not be construed asbeing limited to the descriptions set forth herein. Accordingly, theembodiments are merely described below, by referring to the figures, toexplain aspects of the present description.

The terms used in the specification will be briefly described, andembodiments of the present disclosure will be described in detail.

The terms used in this specification are those general terms currentlywidely used in the art in consideration of functions in regard toembodiments of the present disclosure, but the terms may vary accordingto the intention of those of ordinary skill in the art, precedents, ornew technology in the art. Also, specified terms may be selected, and inthis case, the detailed meaning thereof will be described in thedetailed description of embodiments of the present disclosure. Thus, theterms used in the specification should be understood not as simple namesbut based on the meaning of the terms and the overall description ofembodiments of the present disclosure.

When something “comprises” or “includes” a component, another componentmay be further included unless specified otherwise. Also, the terms“unit” and “module” used herein refer to units specially configured toprocess at least one function or operation, and they may be implementedby hardware, software, or any combination thereof.

Reference will now be made to embodiments of the present disclosure,examples of which are illustrated in the accompanying drawings. In thisregard, the present embodiments may have different forms and should notbe construed as being limited to the descriptions set forth herein.While describing one or more embodiments of the present disclosure,descriptions about drawings that are not related to the one or moreembodiments of the present disclosure are omitted.

Eye gaze tracking is useful in various devices such as laptop computersand smartphones. For example, the eye gaze tracking may control a mousecursor with the eyes in combination with a predetermined gesture and maycontrol presentation and multimedia by the eyes alone. For example, theeye gaze tracking may provide an automatic function such as automaticpage scrolling according to the movement of eyes of a user when readingtext, may profile the user by determining which User Interface (UI) theuser gazes at and which advertisement the user sees, and may assist theuser that is unable to use his hands. Hereinafter, inexpensive andaccurate eye gaze tracking methods and apparatuses according toembodiments of the present disclosure will be described.

FIG. 1 illustrates an eye gaze tracking apparatus according to anembodiment of the present disclosure.

Referring to FIG. 1, an eye gaze tracking apparatus according to anembodiment of the present disclosure includes a display unit (e.g.,monitor) 103 and a photographing unit (e.g., webcam) 105. For example,FIG. 1 illustrates a configuration for tracking a gaze point of a user101 on the display unit 103 by using the photographing unit 105. Asillustrated in FIG. 1, the eye gaze tracking apparatus may beimplemented in a general Personal Computer (PC) equipped with thedisplay unit 103 and the photographing unit 105. However, embodiments ofthe present disclosure are not limited thereto, and for example, the eyegaze tracking apparatus may be implemented in a smartphone equipped witha front camera, a video phone, a smart TeleVision (TV), and similarapparatuses equipped with a display unit and a photographing unit. Thephotographing unit 105 may be disposed in front of the user 101 tophotograph a face of the user 101 that gazes at the display unit 103.

FIG. 2 is a flowchart of a method for acquiring a gaze point of the useron the display unit 103 according to an embodiment of the presentdisclosure.

Referring to FIG. 2, in operation 210, the eye gaze tracking apparatus,which has received an input of an image captured by using thephotographing unit 105, detects a facial feature based on the capturedimage. For example, the facial feature may be, but is not limited to, aneye region, a nose region, or a mouth region. An Active Appearance Model(AAM) may be used to detect the facial feature. Since facial featuredetection may be difficult if the user moves, the eye gaze trackingapparatus may display a message requesting the user not to move. Themessage requesting the user not to move may be provided in the form of asound, an image, text, or the like. For example, along with the outputof a voice, a point may be displayed on the display unit 103 so that theuser may gaze at the point on the display unit 103.

In operation 220, the eye gaze tracking apparatus three-dimensionallymodels the detected facial feature. For example, an AAM may be used torepresent the facial feature. The three-dimensionally modeled facialfeature may be represented in a three-Dimensional (3D) mesh (or 3D grid)and may be, for example, a cylindrical shape or an actual facial shape.Also, an eyeball center may be calculated based on the facial feature,and the calculated eyeball center may be included in thethree-dimensionally modeled facial feature. For example, the eye gazetracking apparatus may detect a position of an iris in a detected eyeregion and calculate a point, which is recessed into an eye by theradius of an eyeball, as an eyeball center. The eye center may becalculated by using an actual eyeball diameter and an actual face sizeof the user or by using statistics thereof. For example, the eye centermay be calculated by using an average eyeball diameter and an averageface size of persons. Embodiments of the present disclosure do notrequire a complex and time-consuming calibration.

Both the eyeball center of the left eye and the eyeball center of theright eye may be calculated. Also, the eye gaze tracking apparatus mayadd the calculated eyeball center as one of the facial features to thethree-dimensionally modeled facial feature.

In operation 230, the eye gaze tracking apparatus tracks thethree-dimensionally modeled facial feature. When a face image of theuser captured by the photographing unit 105 moves, the eye gaze trackingapparatus may detect a movement of the user, track in real time a changein the facial feature that is three-dimensionally modeled in operation220, and reflect the facial feature change, which is caused by themovement of the user, in the three-dimensionally modeled facial feature.Also, the calculated eyeball center may be included in thethree-dimensionally modeled facial feature so as to be tracked togethertherewith. The tracking of the three-dimensionally modeled facialfeature in operation 230 may include determining a facial pose. Thefacial pose may be a pitch, roll, and yaw of the face. The pitch, roll,and yaw of the face may be tracked in consecutive images based on theinitial pitch, roll, and yaw of the face in the initially-capturedimage. The tracking of the three-dimensionally modeled facial featuremay be performed by using an iterative 6D optical flow scheme. Thethree-dimensionally modeled facial feature may be constituted byhundreds of points to ensure higher accuracy.

In operation 240, the eye gaze tracking apparatus detects an iris centerin the face image of the user captured by the photographing unit 105. Inan embodiment, in order to reduce a process of detecting the iriscenter, the iris may be detected by using only the eye region detectedin operation 210. Also, in an embodiment, a position of the iris centeramong the three-dimensionally modeled facial features may be calculatedbased on the iris center detected in the captured face image of theuser. Also, operation 240 may include detecting a position of the irisof the left eye and a position of the iris of the right eye. Dependingon the angle of the face of the user, there may be a difference betweena quality of the detected iris of the left eye and a quality of thedetected iris of the right eye, and a weight may be determined accordingto the qualities.

In operation 250, the eye gaze tracking apparatus acquires an eye gazevector based on the detected iris center and the three-dimensionallymodeled facial feature. For example, referring to FIG. 5, an eye gazevector 540 may be a line that connects a detected iris center 520 and aneyeball center 510 included in a three-dimensionally modeled facialfeature. The eye gaze vector may be acquired for each of the left eyeand the right eye.

In operation 260, the eye gaze tracking apparatus acquires a gaze point530 on the display unit 103 by using the eye gaze vector. In this case,the weight determined according to the qualities of the irises of theleft and right eyes detected in operation 240 may be applied to the eyegaze vector 540. For example, when the quality of the detected iris ofthe left eye is higher than that of the right eye, the gaze point 530 onthe display unit 103 may be acquired by allocating a greater weight tothe eye gaze vector 540 of the left eye. A higher-reliability gaze point530 may be acquired by acquiring the eye gaze point 530 by allocatingthe weight to the eye gaze vector 540. Since a simple geometricoperation is used for the eye gaze vector 540 to detect the gaze point530, actual calibration may not be necessary.

FIG. 3 is a flowchart of an eye gaze detecting method according to anembodiment of the present disclosure.

Referring to FIG. 3, in operation 310, an eye gaze detecting apparatusreceives an image from the photographing unit 105.

In operation 320, the eye gaze detecting apparatus determines whether afacial pose is predetermined.

If the facial pose is not predetermined or if a new facial pose needs tobe determined since there is a great difference between thepredetermined facial pose and the current facial pose, the eye gazedetecting apparatus detects a facial feature in operation 330. Asdescribed above, the facial feature may be an eye region, a mouthregion, or the like. An AAM may be used to detect the facial feature.This operation is used to generate an accurate 3D grid for head posetracking and eye region detection and detect an iris region.

In operation 340, the eye gaze detecting apparatus detects a position ofan iris in the eye region.

In operation 350, the eye gaze detecting apparatus calculates an eyeballcenter based on the detected iris position. Since the eyeball center islocated in a human body in a direction from the iris to the user, theeyeball center may be calculated by using an actual head pose, facesize, and eyeball diameter of the user. When the application of actualvalues is difficult, the eyeball center may be calculated by usingstatistics such as an average face size and eyeball diameter of persons.The eyeball center may be calculated with respect to each of the lefteye and the right eye. Also, a single-frame calibration scheme may beused to easily calculate the eyeball center. That is, the user may berequested to gaze at a predetermined point on the display unit 103 in atleast initial image photographing. Also, in an embodiment, since theuser has only to gaze at one point (not a plurality of points) on thedisplay unit 103, calibration may be performed more easily.

In operation 360, the eye gaze detecting apparatus adjusts a 3D grid tothe detected facial feature and adds the calculated eyeball center tothe 3D grid. For example, the 3D grid may be determined by using afacial feature detector such as an AAM. The 3D grid may have a simplecylindrical shape or may have a more delicate actual facial shape.

Also, before the adjusting of the 3D grid, an initial rotation degree ofthe 3D grid may be determined based on an initial pose of the face, forexample, a pitch, roll, and yaw of the face. Also, an AAM may be used toestimate the initial pose of the face. The calculated eyeball center maybe included in the 3D grid for the detected facial feature.

If the facial pose is not determined in operation 320, or if operation360 is completed, operation 410 of FIG. 4 is performed.

FIG. 4 is a continuation flowchart of FIG. 3, further illustrating theeye gaze detecting method according to an embodiment of the presentdisclosure.

Referring to FIG. 4, in operation 410, the eye gaze detecting apparatustracks the 3D grid by using a continuous 6D optical flow. For example,the 3D grid may be tracked based on each of the captured images. Thatis, a continuous face change in the captured image may be reflected inthe 3D grid. Also, in addition to the 3D grid, the position of theeyeball center included in the 3D grid may be tracked based on eachimage. That is, the eye gaze tracking apparatus may track the positionof the eyeball center according to the continuous face change in thecaptured image. Since the 3D grid, including hundreds of points, istracked in operation 410, the 3D grid may be tracked more accurately.

In operation 420, the eye gaze detecting apparatus determines whetherface tracking is successful. In an embodiment, whether face tracking issuccessful may be determined based on whether a difference between thecaptured face image and the 3D grid is greater than a threshold value.When face tracking is successful, operation 430 is performed, and whenface tracking is not successful, operation 330 of FIG. 3 is performed todetect the facial feature again.

In operation 430, a 3D facial pose is determined based on the 3D gridthat is tracked. In operation 440, an iris center is detected in theimage. The iris center may be detected with respect to each of the lefteye and the right eye. The detecting of the iris center may be performedbased on subpixel accuracy from a general HD webcam in typicalbrightness. That is, the iris center may be detected more accuratelythan the size of a single pixel of the general HD webcam. For example,the detecting of the iris center may be performed by, but not limitedto, using Hough Voting and a Daugman's integro-differential operator. Atechnique used to detect the iris center is not limited thereto, and theiris center may be detected by using any scheme that ensures sufficientsubpixel accuracy and speed.

In operation 450, a 3D iris center is calculated by using the iriscenter detected in the image. For example, the iris center may becalculated on the 3D grid.

In operation 460, an eye gaze vector is calculated by using the facialpose, the eyeball center, and the iris center. The eye gaze vector maybe calculated by a line extending from the eyeball center to the iriscenter and may be calculated with respect to both eyes.

In operation 470, a gaze point on the display unit 103 is calculatedbased on a weighted gaze point. The weighted gaze point may be based onthe weight determined in operation 240. When the gaze point iscalculated by applying the weight, since the weight is allocated tohigher-reliability information, a more accurate gaze point may becalculated. The eye gaze vector is used to calculate the gaze point.

Thereafter, the final gaze point is output at operation 480, and then,operation 310 of FIG. 3 is performed again.

FIG. 6 illustrates an example of the configuration of an eye gazetracking apparatus 600 according to an embodiment of the presentdisclosure. Referring to FIG. 6, the eye gaze tracking apparatus 600 mayinclude a facial feature detecting unit 610, a facial feature trackingunit 620, an iris center detecting unit 630, and an eye gaze trackingunit 640.

The facial feature detecting unit 610 detects a facial feature in acaptured initial image.

The facial feature tracking unit 620 three-dimensionally models thedetected facial feature and tracks the three-dimensionally modeledfacial feature in consecutively captured images.

The iris center detecting unit 630 detects an iris center in theconsecutively captured images.

The eye gaze tracking unit 640 acquires an eye gaze vector based on thetracked three-dimensionally modeled facial feature and the detected iriscenter and acquires a gaze point on a display unit based on the eye gazevector.

FIG. 7 illustrates an example of the operation of the eye gaze trackingapparatus 600 according to an embodiment of the present disclosure.Since the eye gaze tracking method has been described above withreference to FIGS. 2 to 4 in detail, only schematic operations thereofwill be described below.

Referring to FIG. 7, a facial feature detecting unit 720 receives aninitial image from a video sequence 710 input from a photographing unit(e.g., a camera or a video recorder) and detects a facial feature.Although FIG. 7 illustrates that the initial image is received throughan eye gaze tracking unit 740, embodiments of the present disclosure arenot limited thereto. An AAM may be used to detect the facial feature.The detected facial feature is provided to other devices. For example,the detected facial feature may be provided to other devices through theeye gaze tracking unit 740. The facial feature detecting unit 720 doesnot detect a facial feature in subsequent images. However, when failingto track a facial feature in an eye gaze tracking process, the facialfeature detecting unit 720 receives an image again and detects a facialfeature.

The eye gaze tracking unit 740 may control other devices and collectinformation from other devices. When receiving information from thefacial feature detecting unit 720, the eye gaze tracking unit 740 maytransmit the information and consecutive images to a facial featuretracking unit 750. After the images are transmitted to the facialfeature tracking unit 750, a facial feature may be tracked from each ofthe consecutive images and the tracked facial feature may be transmittedto the eye gaze tracking unit 740.

The facial feature tracking unit 750 tracks a facial feature based onthe received information and the consecutive images. The facial featuretracking unit 750 may also track an eye region and an eyeball center.When a difference that is greater than a threshold value exists betweenthe previous image and the subsequent image, the facial feature trackingunit 750 may determine that the facial feature tracking is unsuccessful.When the facial feature tracking is unsuccessful, the facial featuretracking unit 750 may notify this fact to the eye gaze tracking unit740. In this case, the eye gaze tracking unit 740 may control the facialfeature detecting unit 720 to detect a facial feature again. When thefacial feature tracking is successful, the facial feature tracking unit750 may provide an eye position to an iris center detecting unit 760through the eye gaze tracking unit 740.

The iris center detecting unit 760 detects a subpixel iris center,wherein detecting a subpixel iris center is described in more detailbelow.

Upon detecting information, the eye gaze tracking unit 740 calculates aneye gaze vector. The eye gaze vector may be calculated with respect toboth eyes, and a weight may be applied to the eye gaze vector accordingto the iris detection quality. As described above, the eye gaze trackingunit 740 may calculate a gaze point on the display unit based on theweighted eye gaze vectors and the 3D position of the display unit. Also,the calculated gaze point may be output as a final result 730.

Facial Feature Tracking

In a basic algorithm for facial pose tracking in facial featuretracking, an optical flow change for a 3D face model may be used. The 3Dface model may be a semicylinder. The 3D face model may be tracked by ageneral 6D Lukas-Kanade approach. The 6D Lukas-Kanade approach will bedescribed below in brief.

It is assumed that an image I(u,t) is observed at a time t. Herein,u=(u,v) is a pixel in an image. It follows that u shifts to u′=F(u,μ) attime (t+1). Herein, μ is a motion parameter vector and may be obtainedby Equation 1.

μ=[ω_(x), ω_(y), ω_(z), t_(x), t_(y), t_(z)]  Equation 1

F(u,μ) is a parametric motion model and maps u to a new position u′.When there is no change in illumination, Equation 2 will be satisfied.

I(F(u, μ),t+1)=I(u,t)   Equation 2

When a Lucas-Kanade method is applied to Equation 2 and a Taylorexpansion is used, a motion vector formed of a Taylor expansionexpressed as Equation 3 may be calculated.

$\begin{matrix}{\mu = {{- \left( {\sum\limits_{\Omega}\left( {{w\left( {I_{u}F_{\mu}} \right)}^{T}\left( {I_{u}F_{\mu}} \right)} \right)} \right)^{- 1}}{\sum\limits_{\Omega}\left( {w\left( {I_{t}\left( {I_{u}F_{\mu}} \right)}^{T} \right)} \right)}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

Herein, I_(t) is a temporal image gradient, I_(u) is a spatial imagegradient, F_(μ) is a partial derivative of F with respect to μ at μ=0,and w is weights of pixels tracked within a range of [0,1].

An image projection u of X=[x,y,z, 1]T under a perspective projection ata time (t+1) is expressed as Equation 4.

$\begin{matrix}{{u\left( {t + 1} \right)} = {{\begin{bmatrix}{x - {y\omega}_{z} + {z\omega}_{y} + t_{x}} \\{{x\omega}_{z} + y - {z\omega}_{x} + t_{y}}\end{bmatrix} \cdot \frac{f_{L}}{{- {x\omega}_{y}} + {y\omega}_{x} + 2 + t_{z}}}(t)}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

Herein, f_(L), is a focal length.

Equation 4 is a parametric motion model F used in Equation 2. Also,F_(μ) at μ=0 is expressed as Equation 5.

$\begin{matrix}{F_{\mu {{\mu = 0}}} = {{\begin{bmatrix}{- {xy}} & {x^{2} + y^{2}} & {- {yz}} & z & 0 & {- x} \\{- \left( {x^{2} + z^{2}} \right)} & {xy} & {xz} & 0 & z & {- y}\end{bmatrix} \cdot \frac{f_{L}}{z^{2}}}(t)}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

After algorithm iteration, an incremental transformation is calculatedby using μ and may be configured to obtain a final transformationmatrix. A new facial pose may be calculated from a combination of acurrent transformation and a previous pose.

Eyeball Center Detection and Tracking

Eyeball center detection may be performed at an initial stage. The usermay be requested to gaze at a camera straight in the face (or to gaze atanother point of known coordinates). After detecting a face and an eyeregion, the facial feature detecting unit 720 detects an accurate iriscenter. An eyeball center position with respect to a given eyeballradius r and a 3D iris center position p_(iris)(x,y,z) is defined asEquation 6.

P _(eye center) =P _(iris)(x,y,z)+(0,0r)   Equation 6

Herein, P_(eye center) denotes an eyeball center.

That is, P_(eye center) is defined such that the eyeball center islocated at a point that is shifted by the eyeball radius from the pupilalong an axis passing through the camera and the iris center.

The absolute position of the eyeball center changes in consecutiveimages. However, the relative position of the eyeball center withrespect to the entire face is fixed. This may be seen from FIG. 8. Thatis, points in circles of reference numerals 810, 820, 830, and 840represent eyeball centers, and it may be seen that the absolute positionof the eyeball center changes according to the movement of the face.However, it may be seen that the eyeball center is always constant withrespect to the face.

In a 6D Lukas-Kanade approach for tracking a 3D face model, severalalgorithm modifications are performed. Also, along with the modificationof an original algorithm, the face is modeled in a 3D grid.

First, the 3D grid may be implemented in a semicylinder of 20×20 points.Also, for example, the 3D grid may be implemented in a denser 3D grid of40×40 points. In this case, the accuracy of tracking increases, but theperformance thereof is degraded. Also, the face may be modeled in acylindrical shape or in an actual facial shape. Also, thethree-dimensionally modeled face may be modeled as a predefined model,but it may be applied according to the actual face of each user by usinga predetermined calibration scheme (e.g., structure-from-motion).

The positions of points are not uniformly disposed. In an initial stage,a face region may be divided into equal squares of 20×20. However, ineach square, the best points to track (i.e., the edges of the square)may be selected. This technique may use, for example, a Flock ofTrackers approach.

A weight of each point may be calculated by Equation 7 as a differencebetween the value of a subpixel image at a given point in a currentimage and the value of the point in a previous image.

$\begin{matrix}{w = ^{- \frac{{I_{c} - I_{p}}}{\alpha}}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

Herein, I_(c) is a projection of a point in the current image, I_(p) isa projection of a point in the previous image, and α is a definedcomponent that is experimentally calculated. A gradient weight or apixel density weight may not be considered. An image pyramid may beuseful up to a resolution of about 320×240.

2D Iris Center Detection

The detection of the iris center is based on the modification of twoalgorithms, which have been described as the radial symmetry transformand the Daugman's integro-differential operator. The first algorithm mayprovide approximate detection of the iris center, and the secondalgorithm may increase the accuracy of the first algorithm.

Referring to FIG. 9, approximate detection of the iris center is basedon a voting technique, similar to a Circle Hough Transform (CHT). Theapproximate iris detection algorithm is based on the premise that theedge of the iris is a turning point between a bright inner region of theiris and a dark inner region of the iris. The accurate radius of theiris is not known, but the potential range of the iris (e.g., about 7pixels to about 13 pixels) may be estimated based on the size of theface. The left and right edges of the iris are considered because theupper and lower edges of the iris may be covered by the skin around theeye. Thus, the edges of the iris to be considered are vertical portionscorresponding to reference numerals 910 and 920, as illustrated in FIG.9.

An approximate iris center detection algorithm is performed through thefollowing process: First, for an image having a width w and a height h,a 2D w^(×)h voting binary table H is generated and it is completelyfilled with ‘0’. Then, a first derivative of x and y is calculated withrespect to each of image pixels d_(x) and d_(d). The size of the edge ofeach pixel is calculated by Equation 8.

M=√{square root over (d_(x) +d _(y))}  Equation 8

The edge vector of each pixel is calculated by Equation 9.

v=[d _(x) /M; d _(y) /M]  Equation 9

When a size M of each pixel is equal to or greater than a predeterminedthreshold value, a gradient direction is sufficiently vertical. When thevector is directed from a bright side to a dark side,

a.) for each considered radius r, an iris center estimated as a pointconcluded from the movement from a considered point is calculated by avector [x′, y′]=v*r.

b.) each considered radius r, a corresponding H binary [x′, y′] isincreased by an M value of a pixel.

H binary cells corresponding to all pixels are divided by an averageimage value of the points closest to the points of pixels. Since thedividing of the H binary cells corresponding to all pixels isunfavorable for a high-brightness pupil region located in the middle ofthe iris, the pupil region should be dark. For example, GaussianBlurring is used to expand the H binaries. A point of the H binarieshaving the highest value is selected as an approximate iris center.

The voting technique for approximate iris center detection mayseparately generate H tables for the considered radius and may selectthe highest value among all responses. Accordingly, the accuracy of thealgorithm may be increased.

Accurate iris center detection is based on a Daugman'sintegro-differential operator. An iris model is generated by selecting64 or 32 points (e.g., pixels) at an iris boundary corresponding to thereference numerals 910 and 920 of FIG. 9. In this case, horizontalportions of the iris boundary (other than the portions corresponding tothe reference numerals 910 and 920) is disregarded because they may becovered by the skin around the eye as described above. This algorithmdepends on the fact that there is a very high color contrast at the irisboundary. When there is no strong light reflection, a very high colorcontrast exists at the iris boundary regardless of the color of theiris.

In an embodiment, accurate iris center detection may be performed by asubpixel scheme at a higher resolution than the pixel. A differentialfunction response is calculated for all parameter substitutionsapproaching approximate estimation.

In an embodiment, anchoring points (x,y) are, for example, about +−3pixels in x and y directions at intervals of about 0.25 pixel. A radiusr is, for example, about +−2 pixels at intervals of about 1 pixel. Avertical stretch s is a ratio between the maximum height of the iris andthe maximum weight. For example, the vertical stretch s may be used toallow the imitation of the simplest perspective projection as an ellipsewith coefficients ranging from about 0.9 to about 1.1 at intervals ofabout 0.05.

A differential function for each set of parameters x, y, r, and s may becalculated by generating an ellipse with parameters that are separatelydefined with respect to the left and right iris boundaries.

1. Values of pixels are calculated by parameters S_(bright)=(x, y,r+0.5, s) with respect to the outside of an elliptical portion.

2. Values of pixels are calculated by parameters S_(dark)=(x, y, r−0.5,s) with respect to the inside of an elliptical portion.

3. Differential values corresponding to outer and inner ellipticalpoints are calculated. d =S_(dark) S_(bright)

4. The results are sorted in descending order.

5. An average of ¾of all sorted differential values is calculated (byselecting the greatest values).

6. Derived values are returned.

The results from the left and right boundaries of the iris are added togive a final differential result for given parameters. The greatestdifferential value may be selected from all parameter substitutions. Inorder to increase the algorithm performance speed, the algorithm may beperformed from an approximate scheme to an accurate scheme.

3D Iris Center Detection

A 3D position (X_(c), Y_(c), Z_(c)) of the eyeball center, a radius R ofthe eyeball, and a 2D position (u_(x), u_(y)) of the iris may be drawnto a 3D position of the iris. When it is assumed that the iris center islocated at a surface on the eyeball and the eyeball is spherical, asimple projection model may be assumed as Equations 10 to 12.

$\begin{matrix}{R^{2} = {\left( {X - X_{c}} \right)^{2} + \left( {Y - Y_{c}} \right)^{2} + \left( {Z - Z_{c}} \right)^{2}}} & {{Equation}\mspace{14mu} 10} \\{u_{x} = {X\frac{f_{L}}{z}}} & {{Equation}\mspace{14mu} 11} \\{u_{y} = {Y\frac{f_{L}}{z}}} & {{Equation}\mspace{14mu} 12}\end{matrix}$

Herein, X, Y, and Z are 3D coordinates of the iris and f_(I) is a knownfocal length of the camera. Since three Equations 10 to 12 have threeunknown quantities, three unknown quantities may be easily obtained.Sine Equations 10 to 12 are quadratic equations, possible results on aspherical surface may be obtained and a value closest to the camera willbe a final result. Also, a general elliptical (not spherical) equationmay be used to compensate for the anatomical non-uniformity of aneyeball shape of a person and to improve the quality of the algorithm.

Presumptions for Eye Gaze Estimation

In order to obtain a substantial gaze point on the display unit, acoordinate system is calibrated. In order to avoid difficulty in acalibration process, several presumptions may be made on the attributesof the face.

Presumption 1. A distance between both eyes may be fixed at about 6.0 cmto about 6.5 cm or may be calculated in calibration.

Presumption 2. A relation between a model distance z and a face size inpixels may be known (calculated for each camera).

Presumption 3. An eyeball radius may be fixed at about 12 mm and may becalculated for each user during calibration.

Presumption 4. An initial relative shift between a grid surface and aneyeball in a z axis may be known and may be set for each user.

Presumptions 1 and 2 provide approximate estimation of a modelcoordinate system transform with respect to a real coordinate system.Presumption 3 defines a region covered by the movement of all eyes.Presumption 4 may be used to accurately track the eyeball even when thehead moves in any way.

As described above, the embodiments of the present disclosure may beimplemented inexpensively and easily. Also, even when there is amovement of the head, the embodiments of the present disclosure mayoperate without problems; and when there is no obstacle to the movementof the head under good light conditions, an accuracy of at least about1.5 deg. may be achieved in iris direction measurements and an accuracyof at least about 2.5 deg. may be achieved in face directionmeasurements. Also, when there is a small movement of the head, arelative accuracy of about 3 deg. may be allowed to increasecompetitiveness.

It should be understood that the embodiments described therein should beconsidered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each embodimentshould typically be considered as available for other similar featuresor aspects in other embodiments.

While the present disclosure has been shown and described with referenceto the various embodiments thereof, it will be understood by thoseskilled in the art that various changes in form and details may be madetherein without departing from the spirit and scope of the presentdisclosure as defined by the appended claims and their equivalents.

What is claimed is:
 1. A method of tracking an eye gaze, the methodcomprising: detecting a facial feature in a captured initial image;three-dimensionally modeling the detected facial feature; tracking thethree-dimensionally modeled facial feature in consecutively capturedimages; detecting an iris center in the consecutively captured images;acquiring an eye gaze vector based on the tracked three-dimensionallymodeled facial feature and the detected iris center; and acquiring agaze point on a display unit based on the eye gaze vector.
 2. The methodof claim 1, wherein the detecting of the facial feature comprisesdisplaying a message requesting a user to gaze at a certain point of thedisplay unit.
 3. The method of claim 1, wherein the three-dimensionalmodeling of the detected facial feature comprises: calculating aneyeball center based on the detected facial feature; andthree-dimensionally modeling the facial feature including the calculatedeyeball center.
 4. The method of claim 1, wherein the tracking of thethree-dimensionally modeled facial feature comprises: determining afacial pose based on the three-dimensionally modeled facial feature; andtracking the facial pose.
 5. The method of claim 1, wherein thedetecting of the iris center comprises detecting the iris center byusing only an eye region of the three-dimensionally modeled facialfeature.
 6. The method of claim 1, wherein the detecting of the iriscenter comprises calculating a position of the iris center on thethree-dimensionally modeled facial feature.
 7. The method of claim 1,wherein the detecting of the iris center comprises: detecting a positionof an iris of a left eye and a position of an iris of a right eye; anddetermining a weight by estimating a quality of the detected position ofthe iris of the left eye and a quality of the detected position of theiris of the right eye.
 8. The method of claim 7, wherein, the acquiringof the eye gaze vector comprises determining an eye gaze vector of theleft eye and an eye gaze vector of the right eye, and the acquiring ofthe gaze point on the display unit comprises applying the weight basedon the estimated quality of the detected position of the iris of theleft eye and the estimated quality of the detected position of the irisof the right eye to the eye gaze vector of the left eye and the eye gazevector of the right eye, respectively.
 9. The method of claim 1, whereinthe three-dimensional modeling of the detected facial feature comprisesmodeling the facial feature as a three-dimensional grid by using anActive Appearance Model (AAM).
 10. The method of claim 1, wherein thedetecting of the facial feature is performed by using an AAM.
 11. Themethod of claim 3, wherein the calculating of the eyeball center isperformed based on statistics of a facial size and an eyeball diameter.12. An apparatus of tracking an eye gaze, the apparatus comprising: afacial feature detecting unit configured to detect a facial feature in acaptured initial image; a facial feature tracking unit configured tothree-dimensionally model the detected facial feature and to track thethree-dimensionally modeled facial feature in consecutively capturedimages; an iris center detecting unit configured to detect an iriscenter in the consecutively captured images; and an eye gaze trackingunit configured to acquire an eye gaze vector based on the trackedthree-dimensionally modeled facial feature and the detected iris centerand to acquire a gaze point on a display unit based on the eye gazevector.
 13. The apparatus of claim 12, wherein the facial featuredetecting unit is further configured to display a message requesting auser to gaze at a certain point of the display unit when detecting thefacial feature.
 14. The apparatus of claim 12, wherein the facialfeature tracking unit is further configured to calculate an eyeballcenter based on the detected facial feature and to three-dimensionallymodel the facial feature including the calculated eyeball center. 15.The apparatus of claim 12, wherein the facial feature tracking unit isfurther configured to determine a facial pose based on thethree-dimensionally modeled facial feature and to track the facial pose.16. The apparatus of claim 12, wherein the iris center detecting unit isfurther configured to detect the iris center by using only an eye regionof the three-dimensionally modeled facial feature.
 17. The apparatus ofclaim 12, wherein the iris center detecting unit is further configuredto calculate a position of the iris center on the three-dimensionallymodeled facial feature.
 18. The apparatus of claim 12, wherein the iriscenter detecting unit is further configured to detect a position of aniris of a left eye and a position of an iris of a right eye and todetermine a weight by estimating a quality of the detected position ofthe iris of the left eye and a quality of the detected position of theiris of the right eye.
 19. The apparatus of claim 18, wherein the eyegaze tracking unit is further configured to determine an eye gaze vectorof the left eye and an eye gaze vector of the right eye and to acquirethe gaze point on the display unit by applying the weight based on theestimated quality of the detected position of the iris of the left eyeand the estimated quality of the detected position of the iris of theright eye to the eye gaze vector of the left eye and the eye gaze vectorof the right eye, respectively.
 20. The apparatus of claim 12, whereinthe facial feature tracking unit is further configured to model thefacial feature as a three-dimensional grid by using an Active AppearanceModel (AAM).
 21. The apparatus of claim 12, wherein the facial featuredetecting unit is further configured to detect the facial feature byusing an AAM.
 22. The apparatus of claim 14, wherein the facial featuretracking unit is further configured to calculate the eyeball centerbased on statistics of a facial size and an eyeball diameter.
 23. Anon-transitory computer-readable recording medium configured to storecomputer-readable instructions that, when executed by a computer,perform the method of claim 1.